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driving the advancement of recommendation systems. We recognize the press- 
ing hurdles in the field as sparsity and diversity, thereby focusing on intent- 
based models that exploit the additional/auxiliary information by aggregating 
implicit feedback from user-item interactions. We have gone one step further 
by compiling the benchmarks in the field, enabling new researchers to explore 
and innovate at a much more thoughtful and faster pace. 


1. Introduction ”’Grundy” system that asked users questions as an 
input mechanism and classified them (Rich). A 
downside to the proposed method was that users 
of the same group received the same recommen- 
dations, leading to criticism of the approach due 
to the authenticity of the feedback provided by 
the user. Nisbett and Wilson (Nisbett and Wil- 
son) inferred that describing one’s cognitive state 
is improper, and people often fall prey to their 
subconscious stereotypes. In 1992, Belkin and 
Croft (Belkin and Croft) mentioned that the fun- 
damentals of search engines are information filter- 
ing and retrieval systems. Such developments led 

The first recommendation system dates back to the diversification of the field into Collabora- 
to 1979 when Rich and Waters developed the 


With the inevitable rise in available information, 
concocting effective techniques for filtering and 
organizing data is imperative. Enhanced recommen- 
dation systems offer a promising solution, leverag- 
ing advanced reinforcement learning algorithms and 
data-driven insights to deliver personalized, relevant 
recommendations to users (Afsar, Crump, and Far). 
However, with the rapid evolution of technology and 
the increasing availability of data, recommendation 
systems are also rapidly advancing, leading to the 
development of enhanced recommendation systems. 
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tive Filtering (CF) (Breese, Heckerman, and Kadie), 
Content-based Filtering (CbF) (Mooney and Roy), 
and Hybrid systems (Balabanovi¢ and Shoham). 
Further advancements in the CF segment occurred 
in the same year when Goldberg et al. (Goldberg et 
al.) proposed the Tapestry system, which filters the 
information using document relationships. Inspired 
by the Tapestry system, Resnick et al. (Resnick) pre- 
sented the GroupLens architecture in 1994. This 
architecture followed collaborative filters by includ- 
ing ratings and scores aimed at mining reactions 
from news articles. 


Over time, the prevalence of content-based 
recommendation systems came to rise. The main 
aim of CbF systems is the identification of simi- 
lar items. A few content-based systems from the 
germination stage of the field are PSUN (Sorensen 
and Mcelligott), NewsDude (Billsus and Pazzani), 
NewT (Sheth and Maes), LIBRA (Mooney and 
Roy), and PANDORA. Some latest works include 
the JUMP system (Basile), News@hand (Cantador, 
Bellogin, and Castells), Publication Recommenda- 
tion System (D. Wang et al.), LPRS (Dwivedi, Kant, 
and Bharadwaj), LTRS (Nabizadeh et al.), and the 
system proposed by Albatayneh et al. (Albatayneh 
and Ahmad). Content-based systems have also seen 
massive growth, from learning short-term and long- 
term profiles using the Nearest Neighbour and Naive 
Bayesian Classifiers to n-gram and vector repre- 
sentation works. The hybrid systems developed 
from collaborative and content-based filtering tech- 
niques are also worth mentioning. The first hybrid 
systems were FAB (Balabanovi¢é and Shoham), P- 
Tango (Claypool), PTV (Smyth and Cotter), Con- 
tent boosted Collaborative Filtering (Melville et 
al.), and Cinema Screen (Salter and Antonopou- 
los). Hybreed (Hussein et al.), THOR (Sabet and 
Alireza), and the hybrid course recommendation 
system (Perez and Sanguino) are some of the initial 
works in the field. 


With the dominance of neural networks in com- 
puter science, recommendation tasks have benefited 
from deep learning architectures’ excellent pattern 
recognition properties. Ruihui Mu (Mu) conducted 
a comprehensive survey that explains the advantages 
of using deep neural nets for recommendation tasks. 
It is seen that non-linear and non-trivial information 
can be easily captured using deep neural networks, 
which opens up the possibility of solving sparsity 
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issues. The survey (Mu) categorizes the pioneer- 
ing methods into four classes: Autoencoders, Boltz- 
mann machine learning-based models, Recurrent 
Neural Networks (RNNs), and General Adversarial 
Networks (GANs). For collaborative filtering (CF), 
the authors mention state-of-the-art (SOTA) models 
such as ACF (Sedhain), CDAE (Strub, Mary, and 
Philippe), RBM (Salakhutdinov, Mnih, and Hinton), 
RNN-based system (C. Wu et al.), and IRGAN (J. 
Wang). 


Wu et al. (L. Wu et al.) recently conducted a 
more detailed study comparing neural net systems, 
with accuracy as the primary parameter for com- 
parison. The authors mention various models, such 
as NAIS (He et al.), built upon item-based CF and 
achieve significant results on datasets like Movie- 
Lens (Harper, Maxwell, and Konstan) and Pinter- 
est (Geng et al.). The authors also differentiate 
models into four categories: Classical Matrix Fac- 
torization, History Attention, Autoencoder models, 
and Graph Learning Models. With future direc- 
tions to explore long-term and short-term tempo- 
ral features, the authors mention the latest models 
like MA-GNN (Ma et al.), HyperRec (J. Wang et 
al.), IMfOU (Guo, Shi, and C. Liu), KGAT (X. 
Wang), and KGIN (X. Wang et al.). The authors 
also concluded that GNNs capture user-item interac- 
tions at different granularities, making them suitable 
for exploring temporal features using GNN architec- 
tures for sequential learning in recommendations. 


In addition to the advancements in recommen- 
dation systems, there has been significant progress 
in multi-objective optimization. The traditional 
approach to multi-objective optimization involves 
weighing the objectives and finding a single opti- 
mal solution. However, this approach has several 
limitations, such as requiring the user to specify 
the weightings and only finding a single optimal 
solution. In recent years, there has been a shift 
towards using Pareto optimization, which aims to 
find optimal solutions for multiple objectives with- 
out requiring the user to specify weights. Several 
approaches have been proposed for incorporating 
Pareto optimization into recommendation systems, 
including P-MOIA-RS (Chai, Y. Li, and S. Zhu), 
PE-LTR (X. Lin et al.), and AESM? (Zou et al.). 
These approaches attempt to improve recommen- 
dation quality and diversity by taking into account 
numerous objectives such as accuracy, diversity, 
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novelty, and serendipity (Kaminskas and Bridge). 


1.1. Scope and Organization of the Survey 


This paper is organized as follows: Section 2 com- 
prises of the latest models in the field, along with the 
references to relevant datasets used in each work. 
The benchmark metrics are also mentioned in the 
corresponding subsection itself. Section 3 pro- 
vides the compiled view of the popular benchmark 
datasets, and Section 4 gives the conclusion of this 
work. 


2. Recommendation Models 


The use of auxiliary information has been a boon 
to the existence of recommendation systems. Sup- 
plementary information has boosted how models 
roll out user recommendations in various domains. 
Knowledge graphs, reinforcement learning, and 
meta-learning have surpassed the previously known 
limits to recommendations. The reasons for using 
supplementary information are straightforward and 
can be seen from past trends and setbacks in recom- 
mendation systems. 

The trends from the last two decades showcase 
the urgent need to address Sparsity and Diversity. 
Huang et al. (Huang, H. Chen, and Zeng), and 
Papagelis et al. (Papagelis, Plexousakis, and Kut- 
suras) first addressed the sparsity problem in collab- 
orative filtering models. They define sparsity as the 
lack of similarity between users implicitly originat- 
ing due to limited ratings from a given user. They 
emphasize that a given user can only rate so many 
items, resulting in the user-item matrix with many 
empty user-item pairs. The inclusion of additional 
information has been observed to overcome ineffi- 
ciencies in CF models caused by the above cold start 
problem (I Schein et al.). 

The second major hindrance faced by collab- 
orative filtering models is diversity. Although 
Smyth and McClave (Smyth and Mcclave) intro- 
duced diversity as a significant issue in content- 
based systems, the problem is observed in modern 
collaborative filtering models. Diversity or Tempo- 
ral Diversity refers to the characteristic of a model to 
recommend new items to users despite an abundance 
of a specific set of items as training recommenda- 
tions. Lathia (Lathia and Kiritkumar) stated that 
Temporal Diversity is dangerous to collaborative fil- 
tering models as it can lead to loss of interest due to 
stagnant recommendations. Over the years, a solu- 
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tion to both sparsity and diversity is the exploration 
of hidden patterns that can be made more apparent 
to the model using auxiliary information. 

This section presents the latest works contribut- 
ing unique outlooks and tackling diversity and spar- 
sity issues. The following subsections present the 
latest works involving Improved Matrix Factoriza- 
tion (MF), Distance Based models, Graph Con- 
volution Networks (GCNs), Graph Neural Net- 
works (GNNs), Knowledge Graphs (KGs), and 
Causal Graphs (CGs). Some models do not 
include auxiliary information, but these models have 
given intense competition to the ones using aux- 
iliary information. Thus, we categorize the sec- 
tion into more sections than required, enabling 
a more straightforward understanding for future 
researchers. A compilation of all the latest works 
is also presented by Table 1. 


2.1. Matrix Factorization 


Matrix Factorization (MF) has been the most promi- 
nent technique in CF. The root cause of sparsity 
arose in the matrix factorization technique due to the 
lack of user-item pairs. Despite this, many enhanced 
MF-based models have proven efficient, showing 
conventional CF models’ power. 

Lian et al. (Lian et al.) presented Product Quan- 
tized Collaborative Filtering (pQCF) in 2020. The 
pQCF architecture employs a latent space that sep- 
arates users and items into low-dimensional Carte- 
sian dot product subspaces, allowing for the learn- 
ing of clustered representations within each sub- 
space. The authors use the Block Coordinate 
Descent Algorithm for training the model on var- 
ious datasets, such as Yelp, Amazon, Netflix, 
MovieLens, Gowalla, and LastFM. The algorithm 
ensures that convergence is theoretically guaran- 
teed (Tseng). The architecture continues with hash- 
ing and index-based lookups for recommending 
the learned items. Given the efficient indexing, 
the authors achieve a leveled trade-off between the 
accuracy of the model and the low memory cost 
associated with fast item retrieval. An asymmet- 
ric pQCF version dubbed QCF is also presented, 
reaching elevated metrics values. The results for 
QCF on implicit and explicit feedback datasets are 
comparable or elevated for all metrics (Normalized 
Discounted Cumulative Gain@50 [NDCG@S50], 
Recall @50, AUC) compared to SOTA models. 
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2.2. Distance-Based Models 


The recommendations from a model are based on 
the similarity between the user interests. This prop- 
erty opens a new exploration space for the recom- 
mendation models to advance based on distance 
comparison. Distance-based methods have shown 
prevalence in the field over the past decade. How- 
ever, the most prominent work was proposed in 2017 
by Hsieh et al. (Hsieh et al.) named Collaborative 
Metric Learning (CML). 

The approach adopted the triplet loss structure 
and aimed at increasing the distance between neg- 
ative recommendation items and the user while 
reducing the same between positive items and users. 
According to the literature, CML suffered from two 
main problems: it employed a static margin value 
for all users and needed to account for the dissim- 
ilarity between positive and negative items. As Li 
et al. (M. Li et al.) proposed, the first issue arises 
due to the variance in the number of item interac- 
tions for every user. The first issue with CML per- 
tains to the use of a static margin, which can lead to 
a biased model that is less expressive. The second 
challenge concerns the need for improved represen- 
tation of user-item triplets to account for the varying 
degrees of similarity between positive and negative 
items, as opposed to just between positive items and 
users. This can often lead to incorrect recommen- 
dations leading to poor results. To solve the two 
obstacles, the authors introduce Symmetric Metric 
Learning (SML) (M. Li et al.), wherein an adaptive 
margin mechanism enables the model to make better 
decisions. 


1 1 
Lam == Gi pan My + il Yn) 


This adaptive margin can be formulated using Equa- 
tion (1). The resulting loss function is given as 
Equation (2); where u denotes user, (v~, vt] denote 
negative and positive items, d(u,v) represents the 
distance between u and v, (™m,, | denote the adap- 
tive margins for user and item respectively and 1 
is the bound to prevent the margin from being too 
large. 


> cumdes Lacy 
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The authors tested their experiments on the 
Amazon Instant Video Dataset, Yelp, and the 
IMDB Dataset and compared their results using 
three widely used metrics: Precision@K, Hit 
Ratio@K, and Normalized Discounted Cumulative 
Gain (NDCG@K). 


In 2020, Ma et al. (VM. Chen et al.) introduced 
a metric learning model that incorporates a proba- 
bilistic approach and features an adaptive margin. 
The authors targeted Top-K recommendations as 
an output of the architecture. The use of Wasser- 
stein distance for interactions, followed by an adap- 
tive margin generation module, yields a 4.00% to 
22.45% improvement in the Recall @ 10 metric anda 
4.23% to 31.46% improvement in NDCG@ 10. The 
Amazon Books, Electronics, and CDs dataset, the 
GoodReads Comics dataset (Wan and Mcauley), and 
the Gowalla dataset were used for experimentation. 
A bilevel optimization with approximate gradient 
optimization further improves the adaptive margins. 


2.3. Graph Convolution Networks 


In 2020, Sun et al. (Jianing et al.) proposed Neighbor 
interaction aware graph convolution networks (NIA- 
GCN). The NIA-GCN model leverages relational 
information from neighboring nodes to capture the 
heterogeneity of the user-item bipartite graph. They 
utilize three main modules: a Pairwise Neighbor- 
hood Aggregation graph convolution layer (PNA 
layer), Parallel GCNs, and a Cross-Depth Ensem- 
ble layer. The prevalent issue in the field is the 
incorrect consideration of the negative samples as 
unobserved user-item interactions. This issue is 
resolved by the Bayesian Graph Convolutional Neu- 
ral Network framework (BGCF) proposed by Sun et 
al. (Sun et al.). The authors implement the Neigh- 
borhood Copying Graph Generative Model to dis- 
tribute the interaction graph effectively. The authors 
adopt a joint training approach that involves both the 
observed and sampled graphs after the copying and 
redistribution stages. This work aims at resolving 
the uncertainty associated with the user-item inter- 
action graphs. 


To improve the recommendations by GCNs, Wu 
et al. Wu et al. (J. Wu et al.) proposed Self- 
supervised Graph Learning (SGL) in 2021. The 
SGL framework employs an augmentation architec- 
ture to generate different views of each node in the 
interaction graph. The augmented graph is used 
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as an input for contrastive learning, followed by a 
multitasking strategy. The authors used the Yelp, 
Amazon-Book, and Alibaba-iFashion datasets. The 
variations presented by the authors reach double- 
digit improvements for Recall and NDGC metrics. 
In the same year, Xu et al. (Xu) proposed Causal 
Collaborative Filtering (CCF). The work aims at 
estimating causal relations in the user-item interac- 
tions, using do-calculus. The paper describes basic 
notations such as Structural Causal Models, Causal 
Graphs, Intervention, the Causal Effect Rule, Back- 
door Criterion, and the Counterfactual term. The 
causal graph aims to transform the arbitrary rec- 
ommendation system into an unbiased user pref- 
erence estimation, revealing the users’ real prefer- 
ences. Using benchmark datasets such as Moviel- 
Lens, Amazon baby, Yahoo!R3, and Coat Shopping 
dataset, the authors achieve up to 166% improve- 
ment in the Hit@1 metric and up to 46% improve- 
ment in NDGC@ 10 metric. 


In 2022, Zhang et al. (Zhang) proposed IA-GCN: 
Interactive Graph Convolutional Network. Using an 
interaction guidance mechanism, the [A-GCN archi- 
tecture builds explicit interactions between the user 
and item graphs. The root guides the user/item 
tree aggregators to emphasize the importance of 
the children nodes similar to the target user/item 
pair. Such guidance encodes higher-order features 
containing target-specific features, strengthening the 
user’s preference. The results on the Gowalla, Yelp, 
and Amazon-Book datasets are comparable with 
many benchmark results, demonstrating faster train- 
ing than Light-GCN (He). 


After involving intent and enhanced graph 
structure, Lin et al. (Z. Lin et al.) presented 
Neighborhood-enriched = -Contrastive Learning 
(NCL) for improving graph-based applications in 
CF. The core idea of the improvement remains 
the same; improvement of interaction structure, 
although the approach shifts to a new structure- 
contrastive objective that generates positive 
contrastive pairs based on users/items and _ their 
structural neighbors. The idea boils down to 
a multi-task learning strategy that captures the 
representations of homogeneous neighborhoods 
from even layer outputs of the GNN model. This, 
however, introduces noise information in the con- 
trastive pairs. To address the problem of noise, 
the approach employs a_ prototype-contrastive 
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objective that seeks to identify semantic neighbors. 
The authors use the LightGCN (He) model as the 
backbone GNN model. The authors use multiple 
significant benchmark datasets with improvements 
on Recall@K and NDGC@K metrics ranging from 
as low as 0.83% to 12.81%. 


2.4. Graph Neural Networks 


Wang et al. (Xiang et al.) proposed the work on 
Disentangled graph collaborative filtering (DGCF) 
yielding disentangled representations with the help 
of intent-aware graph networks. The architec- 
ture involves the graph disentangling module and 
the independence modeling module. After slicing 
them into chunks, the disentangling module cou- 
ples the user-item vectors with intent. The GNN 
model in the module includes a unique neighbor 
routing mechanism in the propagation path for the 
embeddings. This mechanism distinguishes the 
different importance levels of interactions between 
users and items in the interaction graphs. The 
authors use cross-intent embedding propagation to 
further enhance the interaction graph’s intent learn- 
ing. The second module involves mutual infor- 
mation and a distance correlation term that allow 
the factor-aware representations to be independent. 
The model parameters are optimized using the BPR 
Loss (Rendle). The architecture improves bench- 
mark metrics like Recall and NGDC by approxi- 
mately 4%, 5%, and 12% for the Gowalla, Yelp, and 
Amazon-Book Dataset. This work is beneficial for 
providing insights into the role of users’ intent in 
recommender systems. 


Another important work in the GNN workspace 
was proposed by Yang et al. (Yang et al.) in 
2021. The authors present the enhanced graph learn- 
ing network (EGLN) for collaborative filtering via 
mutual information maximization. According to the 
authors, relying on a fixed graph structure can result 
in negative and unknown positive preferences mixed 
up with users’ unobserved behaviors. To mitigate 
this issue, their model improves upon the user-item 
bipartite graph by adding a residual graph structure 
learned iteratively alongside the node embedding 
graph. the EGLN architecture achieves an approx- 
imate 7% improvement in both Hit Ratio@10 and 
NDGC@10 for the MovieLens-1M and Amazon- 
Video Games Dataset. The authors improve the met- 
rics by 3% and 5%, respectively, for the Pinterest 
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Dataset. 


2.5. Knowledge Graphs 


Sang et al. (Sang et al.) proposed Knowledge Graph 
enhanced Neural Collaborative Filtering with Resid- 
ual Recurrent Network (KGNCF-RNN) in 2022. 
The architecture uses a knowledge graph (KG) to 
extract path embeddings for relations between enti- 
ties in the KG. A path is formulated such that 
= (e1 eT eee —— er | , where e; € 
I,e, € I; (e, €,, e741) is a triplet, and J is the Inter- 
action Graph. Such paths of different lengths can 
be created for every user/item pair and are denoted 
by p;. These path embeddings are fed to a Recur- 
rent Residual Network (RRN) for contextual learn- 
ing. The output embedding obtained from the RRN 
module is given by (his hy,-* Piya This is then 
applied to the self-attention model resulting in Equa- 
tion (3), where f is the attention function. 

f (hi, hj) = Wo (Wah, + Wahi) 


wo"7 


exp(f (hihj)) 
deja e@P(S (hih5)) 


The Correlation matrix or the interaction matrix E? 
is formulated as given in Equation 4. These two 
matrices are passed as an input to a CNN model, 
resulting in feature recommendations. 


Ei; = (3) 


FE? = ky, @ j; = hug (4) 


The authors of KGNCF-RNN improve on the bench- 
mark results on the MovieLens, Book Crossing, and 
LastFM datasets. 

Another significant work from 2022 was pro- 
posed by Wang et al. (X. Wang et al.). The authors 
present Knowledge Graph-based Intent Network 
(KGIN), where they aim to learn the intents behind 
interactions in KGs. They also devise an informa- 
tion aggregation scheme for GNNs for accumulat- 
ing long-range connectivity from relational paths. 
The architecture involves creating an intent graph 
using intent representations obtained from repre- 
sentation learning. In order to remove the redun- 
dancy of intent learned, an independence model- 
ing module is implemented. An aggregation layer 
is devised that first goes over the intent graph and 
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then the original knowledge graph. This allows the 
model to aggregate the knowledge from both graphs, 
improving the capture of relational paths. This work 
also shows a significant contribution in the Amazon- 
Book, Alibaba-iFashion, and LastFM datasets. 


3. Benchmark Datasets 


The following section provides a comprehensive 
overview of the benchmark datasets available in the 
recommendation systems domain. Research and pri- 
vate organizations, including universities, non-profit 
organizations, and product-based companies like 
Amazon and Netflix, have published these datasets. 
The datasets are classified for future researchers 
to quickly understand and pick the dataset that 
best matches their field of study. The list is non- 
exhaustive, although it comprises all the benchmark 
datasets used for experimenting and publishing the 
results of recommendation system models. 


3.1. Movie Datasets 
3.1.1. MovieLens-25M Dataset: 


This is one of the most used movie rating datasets 
provided by GroupLens Research. The dataset com- 
prises approximately 25 million ratings and one mil- 
lion tags collected from 162,000 users and 62,000 
movies. The movies are rated by users and classified 
into various genres. First published by Harper and 
Konstan (Harper, Maxwell, and Konstan) in 2015, 
the dataset is available in sizes 1M, 20M, and 25M. 


3.1.2. Netflix $1M Prize Dataset: 


Netflix released this dataset to motivate researchers 
to develop improvised recommendation models. 
Introduced by Bennett et al. (Bennett and Lanning) 
in 2007, the dataset contains 17,770 movies and rat- 
ings from 480,189 users. The competition encour- 
aged participants to improve their RMSE score for a 
grand prize of 1 million dollars. 


3.1.3. Flixster Dataset: 


Zafarani and Liu (Zafarani and H. Liu) from Ari- 
zona State University published the dataset in 2009. 
The dataset contains 2,523,386 nodes as users from 
the website Flixster, making 9,197,338 edges, indi- 
cating the friendship between users. The dataset is 
available on the ASU Data Repository. 


3.2. Music Datasets 
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3.2.1. Million Song Dataset: 


The Million Song Dataset is a cluster of complemen- 
tary datasets the community contributes for research 
purposes. A collaborative effort by The Echo Nest 
and LabROSA (Bertin-Mahieux), the most famous 
subset is the Last.FM dataset. This song corpus 
includes song tags and similarity attributes for the 
Million Song Dataset, with 522,366 unique tags and 
56,506,688 (track - similar track) pairs. 


3.2.2. Yahoo! Music Dataset: 


This dataset aggregates around 136 thousand songs 
rated by 1.8 million users. The 717 million ratings 
were scraped from the Yahoo! Music Website. The 
songs are collected between 2002 and 2006, with 
genre attributes and genre hierarchy. 


3.3. Product-based Datasets 
3.3.1. Amazon Product Review Dataset: 


First Released in 2014, the Amazon Review Data 
contains reviews from product metadata such as 
description, price, and brand information. Jianmo 
at UCSD (Ni, J. Li, and Mcauley) published the 
dataset with 233.1 million reviews containing rat- 
ings, text, and helpfulness votes, along with reviews 
from product categories like Books, Video-Games, 
CDs, Movies, Beauty, and many more. The per- 
category data is also available for domain-specific 
tasks. 


3.3.2. GoodBooks-10k Dataset: 


Introduced by FastML (Zajac), this book’s dataset 
contains 6 million ratings for the most popular ten 
thousand books. Available on FastML, the dataset 
contains explicit ratings and implicit feedback mark- 
ers for researchers to experiment. The authors have 
also created tags for the books, enabling a more 
extensive metadata study to be conducted. 


3.3.3. Alibaba-iFashion: 

Chen et al. (W. Chen et al.) published the Alibaba- 
iFashion dataset in 2019. Clicks behaviors and 
events from 3.57 million users and 4.75 million 
items make it the largest Fashion Product recom- 
mendation domain dataset. 


3.4. Image and Video Datasets 
3.4.1. Pinterest Dataset: 


Geng et al. (Geng et al.) published this dataset by 
sampling users from 468 categories on Pinterest. 
The pinned images and additional information indi- 
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cating categories from | million users were crawled 
from the Pinterest Website. The dataset is available 
on the Xue @ Alphabeta Website. 


3.4.2. Steam Dataset: 


Pathak et al. (Pathak, Gupta, and Mcauley) pub- 
lished the Steam Video Game Dataset by compiling 
Steam Video Gaming Platform reviews. It contains 
7,793,069 reviews from over 2.5 million users and 
15 thousand items. It has metadata attributes like 
purchases, plays, pricing information, and likes. 


3.4.3. Miscellaneous Datasets 


Given the rise of e-commerce, many other domain- 
specific datasets are available for researchers to 
use. Among these, the most common ones are 
mentioned in this section. The MIcrosoft News 
Dataset (MIND) published by Wu et al. (RF Wu 
et al.) in 2020, is a large-scale dataset captured 
from Microsoft News. The Gowalla Dataset (Cho 
et al.) available on the SNAP Stanford Repository 
is a location-based social networking dataset with 
196,591 nodes and 950,327 edges. The Douban 
Dataset is a high sparsity dataset crawled by Zhu 
et al. (KF Zhu) from the Douban Website. It con- 
tains user and item interactions from various cate- 
gories like movies, books, and music. Similarly, the 
Yelp Dataset contains business reviews, including 
over 200 thousand images and around 1.2 million 
attributes from businesses including parking, hours, 
ambiance and availability. 


4. Conclusion 


This paper presents the evolution of recommenda- 
tion systems, particularly collaborative filtering sys- 
tems. Our focus remains on using auxiliary/implicit 
information and its impact on the existing architec- 
tures. As seen from the recent works, a significant 
improvement is delivered by the systems capturing 
the hidden intent of users. Often, this hidden intent 
is learned using knowledge graphs, node neighbor- 
hood similarity matching, and mutual information. 
For future researchers to understand the field, this 
work can serve as a consolidated, non-exhaustive 
compilation that summarizes the eventual use of 
additional information in the recommender system 
space. 
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TABLE 1. Comparison of Latest Enhanced Recommendation Systems 


Model Architecture Year Algorithms Datasets Metrics 
Type Name 
MF Product 2021 Block Coordinate Yelp2018, NDCG@ KRecall 
Quantized Descent Algorithm, Amazon Review, @KAUC(K=50) 
Collaborative Hashing and Netflix, 
Filtering (pQCF) Indexing MovieLens, 
Gowalla, LastFM 
Symmetric 2020 Adaptive Margin, Amazon Precision @ KHit 
Distance- Metric Learning SML over Instant- Video, Ratio@ KNDCG 
Based (SML) Collaborative Yelp2018, IMDB @K(K=5,10) 
Metric Learning 
Probabilistic 2020 Adaptive Margin, Amazon Review, NDCG@KRecall 
Metric Learning Probabilistic GoodReads @K(K=50) 
Approach, Bilevel § Comics, Gowalla 
Optimization using 
Approximate 
Gradient Descent 
Neighbor 2020 Pairwise Gowalla, Amazon NDCG@KRecall 
Interaction Neighborhood Review, Industrial @KAUC(K=20) 
GCN Aware Graph Aggregation Data 
Convolution Convolution Layer, 
Networks Parallel GCNs, and 
(NIA-GCN) a Cross-Depth 
Ensemble Layer. 
Bayesian Graph 2020 Neighborhood Amazon Review NDCG@KRecall 
Convolutional Copying Graph @KSerendipity 
Neural Generative Model @K(K=10,20) 
Networks using Joint Training 
(BGCF) 
Self-supervised 2021 GCN with Yelp2018, NDCG@KRecall 
Graph Learning Multi-task learning © Amazon-Book, @K(K=20) 
(SGL) and Contrastive Alibaba-iFashion 
Learning 
Causal 2021 Structural Causal MovieLens- Hit 
Collaborative Models, 100K, Amazon Ratio@ INDCG@ 10 
Filtering (CCF) Intervention, Baby, Yahoo R3, 
Backdoor Criterion, Coat Shipping 
Lagrange 
optimization for 
counterfactual 


reasoning in 
discrete space 
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Table I continued 


GNN 


KG 


Interactive 
Graph 
Convolutional 
Network 
([A-GCN) 
Disentangled 
Graph 
Collaborative 
Filtering 
(DGCF) 
Enhanced Graph 
Learning 
Network 
(ELGN) 


Neighborhood- 
enriched 
Contrastive 
Learning (NCL) 


Knowledge 
Graph enhanced 
Neural 
Collaborative 
Filtering with 
Residual 
Recurrent 
Network 
(KGNCF-RNN) 
Knowledge 
Graph-based 
Intent Network 
(KGIN) 


2022 


2020 


2021 


2022 


2021 


2021 


2 Tree GCN 
approach with 
guiders for focusing 
on target specific 
features 

Graph 
Disentangling, 
Independence 
Modeling, 
BPRLoss 
Similarity 
Calculation, 
Residual Graph, 
Node Embedding 
Learning 
Contrastive 
Learning with 
Structural and 
Semantic 
Neighbors 

KG, RRN, 
Noise-Contrastive 
Estimation, 
Convolution NCF 


KG based entity- 
relation-entity 
triplets + 
User-intent-item 
triplets using intent 
representation, BPR 
Loss for 
optimization) 


Gowalla, 
Yelp2018, 
Amazon-Book 


Yelp2018, 
Gowalla, Amazon 
Book 


MovieLens-1M, 
Amazon-Video 
Games, Pinterest 


Yelp2018, 
Amazon-Book 


MovieLens, Book 
Crossing, LastFM 


Amazon-Book, 
LastFM, 
Alibaba-iFashion 


NDCG@ KRecall 
@K(K=20) 


NDCG@K’2Recall 
@K(K=20) 


Hit Ratio@ KNDCG 
@K(K=5,10,15,20,25,30) 


NDCG@ KRecall @ K(K=10, 


20, 50) 


NDCG @ KRecall @ K(K=20) 


AUCF1 
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Further, it is clear from the challenges researchers 
face in the field that diversity and sparsity can hinder 
recommender systems’ performance. Such issues 
can lead to redundant recommendations or a com- 
plete lack of reliability in the system. For future 
architectures to understand the implicit interests of 
users, graph networks with supporting information 
can prove to be highly efficient, both in terms of 
results and learning capabilities. These issues will 
increase given the vast amount of data flowing into 
the databases, although an effort to reduce them can 
be made by following the current works presented in 
the paper. 


References 


Afsar, M Mehdi, Trafford Crump, and Behrouz Far. 
“Reinforcement Learning based Recommender 
Systems: A Survey”. ACM Computing Surveys 
55.7 (2023): 1-38. 


Albatayneh, Naji and Ahmad. “Utilizing learners’ 
negative ratings in semantic content-based rec- 
ommender system for e-learning forum”. Journal 
of Educational Technology & Society 21 (2018): 
112-125. 


Balabanovic, Marko and Yoav Shoham. “Fab: 
content-based, collaborative recommendation”. 
Communications of the ACM 40 (1997): 66-72. 


Basile, Pierpaolo. “The JUMP project: Domain 
Ontologies and Linguistic Knowledge@ Work”. 
SWAP (2007). 


Belkin, Nicholas J and W Bruce Croft. “Information 
filtering and information retrieval: Two sides of 
the same coin”. Communications of the ACM 35 
12 (1992): 29-38. 


Bennett, J and S Lanning. “The netflix prize”. Pro- 
ceedings of KDD cup and workshop. 2007. 35— 
a3. 


Bertin-Mahieux, T. “The million song dataset in 
Proceedings of the 12th International Society 
for Music Information Retrieval Conference”. 
The million song dataset in Proceedings of the 
12th International Society for Music Information 
Retrieval Conference. 2011. 591-596. 


Billsus, Daniel and Michael J Pazzani. “A Hybrid 
User Model for News Story Classification”. CISM 


International Research Journal on Advanced Science Hub (IRJASH) 


2023, Vol. 05, Issue 05S May 


International Centre for Mechanical Sciences 
(1999): 99-108. 

Breese, John S, Carl David Heckerman, and Kadie. 
“Empirical analysis of predictive algorithms for 
collaborative filtering”. (2013). 


Cantador, Ivan, Alejandro Bellogin, and Pablo 
Castells. “News@hand: A Semantic Web 
Approach to Recommending News”. Lecture 
Notes in Computer Science (2008): 279-283. 


Chai, Zhengyi, Yalun Li, and Sifeng Zhu. “P- 
MOIA-RS: a multi-objective optimization and 
decision-making algorithm for recommendation 


systems”. Journal of Ambient Intelligence and 
Humanized Computing 12.1 (2021): 443-454. 


Chen, Ma, et al. “Probabilistic Metric Learning with 
Adaptive Margin for Top-K Recommendation”. 
Proceedings of the 26th ACM SIGKDD Inter- 
national Conference on Knowledge Discovery & 
Data Mining (2020). 


Chen, Wen, et al. “POG: Personalized Outfit Gen- 
eration for Fashion Recommendation at Alibaba 
iFashion’’. Proceedings of the 25th ACM SIGKDD 
International Conference on Knowledge Discov- 
ery & Data Mining (2019). 


Cho, et al. “Friendship and mobility: user movement 
in location-based social networks”. Proceedings 
of the 17th ACM SIGKDD international confer- 
ence on Knowledge discovery and data mining 
(2011). 


Claypool, Mark. “Combing content-based and 
collaborative filters in an online newspaper”. 
Proc. of Workshop on Recommender Systems- 
Implementation and Evaluation (1999). 


Dwivedi, Pragya, Vibhor Kant, and Kamal K 
Bharadwaj. “Learning path recommendation 
based on modified variable length genetic algo- 
rithm”. Education and Information Technologies 
23.2 (2018): 819-836. 


Geng, Xue, et al. “Learning Image and User Fea- 
tures for Recommendation in Social Networks”. 
2015 IEEE International Conference on Com- 
puter Vision (ICCV) (2015). 


Goldberg, David, et al. “Using collaborative filter- 
ing to weave an information tapestry”. Communi- 
cations of the ACM 35.12 (1992): 61-70. 


550 


Enhanced Recommendation Systems: A Survey on the Impact of Auxiliary Information 


Guo, Xueliang, Chongyang Shi, and Chuanming 
Liu. “Intention Modeling from Ordered and 
Unordered Facets for Sequential Recommenda- 
tion”. Proceedings of The Web Conference 2020 
(2020). 


Harper, F, Joseph A Maxwell, and Konstan. “The 
movielens datasets: History and context”. Acm 
transactions on interactive intelligent systems 


(tiis) 5 (2015): 1-19. 


He, Xiangnan. “Lightgcn: Simplifying and power- 
ing graph convolution network for recommenda- 
tion”. Proceedings of the 43rd International ACM 
SIGIR conference on research and development in 
Information Retrieval. 2020 (2020). 


He, Xiangnan, et al. “NAIS: Neural Attentive Item 
Similarity Model for Recommendation”. JEEE 
Transactions on Knowledge and Data Engineer- 
ing 30.12 (2018): 2354-2366. 


Hsieh, Cheng-Kang, et al. “Collaborative Metric 
Learning”. Proceedings of the 26th International 
Conference on World Wide Web (2017). 


Huang, Zan, Hsinchun Chen, and Daniel Zeng. 
“Applying associative retrieval techniques to alle- 
viate the sparsity problem in collaborative filter- 
ing”. ACM Transactions on Information Systems 
22.1 (2004): 116-142. 


Hussein, Tim, et al. “Hybreed: A software frame- 
work for developing context-aware hybrid rec- 
ommender systems”. User Modeling and User- 
Adapted Interaction 24.1-2 (2014): 121-174. 


I Schein, Andrew, et al. “Methods and metrics for 
cold-start recommendations”. Proceedings of the 
25th annual international ACM SIGIR confer- 
ence on Research and development in information 
retrieval (2002). 


Jianing, Sun, et al. “Neighbor Interaction Aware 
Graph Convolution Networks for Recommenda- 
tion”. Proceedings of the 43rd International ACM 
SIGIR Conference on Research and Development 
in Information Retrieval (2020). 


Kaminskas, Marius and Derek Bridge. “Diversity, 
serendipity, novelty, and coverage: a survey and 
empirical analysis of beyond-accuracy objectives 
in recommender systems”. ACM Transactions on 


2023, Vol. 05, Issue 05S May 


Interactive Intelligent Systems (TiS) 7.1 (2016): 
1-42. 


Lathia, Neal and Kiritkumar. “Evaluating collabora- 
tive filtering over time”. Diss (2010). 


Li, Mingming, et al. “Symmetric Metric Learn- 
ing with Adaptive Margin for Recommendation”. 
Proceedings of the AAAI Conference on Artificial 
Intelligence 34.04 (2020): 4634-4641. 


Lian, Defu, et al. “Product Quantized Collaborative 
Filtering”. IEEE Transactions on Knowledge and 
Data Engineering 33.9 (2021): 3284-3296. 


Lin, Xiao, et al. “A pareto-efficient algorithm for 
multiple objective optimization in e-commerce 
recommendation”. Proceedings of the 13th ACM 
Conference on Recommender Systems (2019). 


Lin, Zihan, et al. “Improving Graph Collabora- 
tive Filtering with Neighborhood-enriched Con- 
trastive Learning”. Proceedings of the ACM Web 
Conference 2022 (2022). 


Ma, Chen, et al. “Memory Augmented Graph Neu- 
ral Networks for Sequential Recommendation’. 
Proceedings of the AAAI Conference on Artificial 
Intelligence 34.04 (2020): 5045-5052. 


Melville, et al. “Content-boosted collaborative filter- 
ing for improved recommendations”. Aaai/iaai 23 
(2002): 187-192. 


Mooney, Raymond J and Loriene Roy. “Content- 
based book recommending using learning for text 
categorization”. Proceedings of the fifth ACM 
conference on Digital libraries (2000). 


Mu, Ruihui. “A Survey of Recommender Systems 
Based on Deep Learning”. IEEE Access 6 (2018): 
69009-69022. 


Nabizadeh, Amir Hossein, et al. “Adaptive learn- 
ing path recommender approach using auxiliary 
learning objects”. Computers & Education 147 
(2020): 103777—103777. 


Ni, Jianmo, Jiacheng Li, and Julian Mcauley. “Justi- 
fying Recommendations using Distantly-Labeled 
Reviews and Fine-Grained Aspects”. Proceedings 
of the 2019 Conference on Empirical Methods in 
Natural Language Processing and the 9th Inter- 
national Joint Conference on Natural Language 
Processing (EMNLP-IJCNLP) (2019). 


Navansh Goel and Suganeshwari G 


Nisbett, Richard E and Timothy D Wilson. “Telling 
more than we can know: Verbal reports on men- 
tal processes.” Psychological Review 84.3 (1977): 
231-259. 


Papagelis, Manos, Dimitris Plexousakis, and 
Themistoklis Kutsuras. “Alleviating the Sparsity 
Problem of Collaborative Filtering Using Trust 
Inferences”. Lecture Notes in Computer Science 
(2005): 224-239. 


Pathak, Apurva, Kshitiz Gupta, and Julian Mcauley. 
“Generating and Personalizing Bundle Recom- 
mendations on jijSteamj/ij”. Proceedings of 
the 40th International ACM SIGIR Conference 
on Research and Development in Information 
Retrieval (2017). 


Perez, Juan Camilo and Sanguino. “A course hybrid 
recommender system for limited information sce- 
narios”. Journal of Educational Data Mining 14 
(2022): 162-188. 


Rendle, Steffen. “BPR: Bayesian personalized rank- 
ing from implicit feedback”. BPR: Bayesian per- 
sonalized ranking from implicit feedback (2012). 


Resnick, Paul. “Grouplens: An open architecture for 
collaborative filtering of netnews”. Proceedings 
of the 1994 ACM conference on Computer sup- 
ported cooperative work (1994). 


Rich, Elaine. “User modeling via stereotypes”. Cog- 
nitive science 3 (1979): 80012-80021. 


Sabet, Javadian and Alireza. “THOR: A Hybrid 
Recommender System for the Personalized Travel 
Experience”. Big Data and Cognitive Computing 
6 (2022): 131-131. 


Salakhutdinov, Ruslan, Andriy Mnih, and Geof- 
frey Hinton. “Restricted Boltzmann machines for 
collaborative filtering”. Proceedings of the 24th 
international conference on Machine learning 


(2007). 


Salter, J and N Antonopoulos. “CinemaScreen Rec- 
ommender Agent: Combining Collaborative and 
Content-Based Filtering”. [EEE Intelligent Sys- 
tems 21.1 (2006): 35-41. 


Sang, Lei, et al. “Knowledge Graph enhanced Neu- 
ral Collaborative Filtering with Residual Recur- 
rent Network”. Neurocomputing 454 (2021): 
417-429. 


2023, Vol. 05, Issue 05S May 


Sedhain, Suvash. ‘“Autorec: Autoencoders meet col- 
laborative filtering’. Proceedings of the 24th 
international conference on World Wide Web 
(2015). 


Sheth, B and P Maes. “Evolving agents for per- 
sonalized information filtering”. Proceedings of 
9th IEEE Conference on Artificial Intelligence for 
Applications (1993). 


Smyth, B and P Cotter. “A personalised TV listings 
service for the digital TV age”. Knowledge-Based 
Systems 13.2-3 (2000): 53-59. 


Smyth, Barry and Paul Mcclave. “Similarity vs. 
diversity”. Vancouver, BC, Canada; Berlin Hei- 
delberg: Springer, 2001. 


Sorensen, Humphrey and Michael Mcelligott. 
“PSUN: a profiling system for Usenet news”. Pro- 
ceedings of CIKM 95 (1995). 


Strub, Florian, Jeremie Mary, and Preux Philippe. 
“Collaborative filtering with stacked denoising 
autoencoders and sparse inputs”. NIPS workshop 
on machine learning for eCommerce (2015). 


Sun, Jianing, et al. “A Framework for Recommend- 
ing Accurate and Diverse Items Using Bayesian 
Graph Convolutional Neural Networks”. Pro- 
ceedings of the 26th ACM SIGKDD International 
Conference on Knowledge Discovery & Data 
Mining (2020). 


Tseng, Paul. “Convergence of a Block Coordinate 
Descent Method for Nondifferentiable Minimiza- 
tion”. Journal of Optimization Theory and Appli- 
cations 109.3 (2001): 475-494. 


Wan, Mengting and Julian Mcauley. “Item recom- 
mendation on monotonic behavior chains”. Pro- 
ceedings of the 12th ACM Conference on Recom- 
mender Systems (2018). 


Wang, Donghui, et al. “A content-based recom- 
mender system for computer science publica- 
tions”. Knowledge-Based Systems 157 (2018): 1- 
9. 


Wang, Jianling, et al. “Next-item Recommenda- 
tion with Sequential Hypergraphs”. Proceedings 
of the 43rd International ACM SIGIR Confer- 
ence on Research and Development in Informa- 
tion Retrieval (2020). 


International Research Journal on Advanced Science Hub (IRJASH) 552 


Enhanced Recommendation Systems: A Survey on the Impact of Auxiliary Information 


Wang, Jun. “Irgan: A minimax game for unify- 
ing generative and discriminative information 
retrieval models”. Proceedings of the 40th Inter- 
national ACM SIGIR conference on Research and 
Development in Information Retrieval (2017). 


Wang, Xiang. “Kgat: Knowledge graph attention 
network for recommendation’. Proceedings of the 
25th ACM SIGKDD international conference on 
knowledge discovery & data mining (2019). 


Wang, Xiang, et al. “Learning Intents behind Inter- 
actions with Knowledge Graph for Recommenda- 
tion”. Proceedings of the Web Conference 2021 
(2021). 


Wu, Caihua, et al. “Recurrent neural network based 
recommendation for time heterogeneous feed- 
back”. Knowledge-Based Systems 109 (2016): 
90-103. 


Wu, Fangzhao, et al. “MIND: A Large-scale Dataset 
for News Recommendation”. Proceedings of the 
58th Annual Meeting of the Association for Com- 
putational Linguistics (2020). 


Wu, Jiancan, et al. “Self-supervised Graph Learn- 
ing for Recommendation’. Proceedings of the 
44th International ACM SIGIR Conference 
on Research and Development in Information 
Retrieval (2021). 


Wu, Le, et al. “A Survey on Accuracy-oriented Neu- 
ral Recommendation: From Collaborative Filter- 
ing to Information-rich Recommendation”. [EEE 
Transactions on Knowledge and Data Engineer- 
ing (2022): 1-1. 

Xiang, Wang, et al. “Disentangled Graph Collabora- 
tive Filtering”. Proceedings of the 43rd Interna- 
tional ACM SIGIR Conference on Research and 
Development in Information Retrieval (2020). 


Yang, Yonghui, et al. “Enhanced Graph Learning for 
Collaborative Filtering via Mutual Information 


2023, Vol. 05, Issue 05S May 


Maximization”. Proceedings of the 44th Interna- 
tional ACM SIGIR Conference on Research and 
Development in Information Retrieval (2021). 


Zhang, Yinan. “I[A-GCN: Interactive Graph Con- 
volutional Network for Recommendation’. JA- 
GCN: Interactive Graph Convolutional Network 
for Recommendation (2022). 


Zhu, Feng. “Dtcdr: A framework for dual-target 
cross-domain recommendation”. Proceedings of 
the 28th ACM International Conference on Infor- 
mation and Knowledge Management (2019). 


Zou, Xinyu, et al. “Automatic Expert Selection for 
Multi-Scenario and Multi-Task Search”. Proceed- 
ings of the 45th International ACM SIGIR Confer- 
ence on Research and Development in Informa- 
tion Retrieval (2022). 


@) © Navansh Goel et al. 
(c) = 2023 Open Access. This 


article is distributed 
under the terms of the Creative Com- 
mons. Attribution 4.0 International License 


(http://creativecommons.org/licenses/by/4.0/), 
which permits unrestricted use, distribution, and 
reproduction in any medium, provided you give 
appropriate credit to the original author(s) and the 
source, provide a link to the Creative Commons 
license, and indicate if changes were made. 


Embargo period: The article has no embargo 
period. 


To cite this Article: , Navansh Goel, and Sug- 
aneshwari G . “Enhanced Recommendation Sys- 
tems: A Survey on the Impact of Auxiliary 
Information . International Research Journal on 
Advanced Science Hub 05.05S May (2023): 541- 
553. http://dx.doi.org/10.47392/irjash.2023.S073 


